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WINDOWS API TRAPPING SYSTEM 

Cross-Reference to Rdated AppHonrinnc 

This application is based on U.S. provisional patent application No. 60/028,339, 
filed on October 1 1, 1996. 

5 Background of The Tnv«Mitinn 

1. Field of the Invention 

This application relates to the field of software and more particulariy to the fidd of 
managing aspects of an interface between software and the underlying operating system. 

2. Descriprion of RelatftH Aft 

*® Many conventional operating systems provide a formalized Application 

Programming Interfece (API) that aUows application programmers to make calls to 
software routines that perform a variety of system-wide functions. Using the API 
fiidlitates writing appUcation programs by decreasing the amount of code that appHcation 
programmers need to provide and, at the same time, providing standardization of routines 

15 that are used by many of the applications. 

However, in some instances, it is necessary to modify an API call in order to, for 
example, perform spedalized functions that are not provided by the operating system or to 
keep track of certain types of API calls. For some operating systems, it is not difficult to 
intercept and monitor API caUs. For example, under MS-DOS, most APIs use interrupts 

20 and are thus easy to intercept. In other instances, intercepting and trapping API calls can 

be challenging for an application programmer. For example, in some Microsoft Windows 
Environments (Wmie and Win32), tiie APIs use exported functions which are connected 
to the appUcation at application load time via static or dynamic linkirtg. This linkage 
process is done by the internal OS routines and undocumented data stnictures vAich are 

25 usually inaccessible to outside (non-Microsoft) software developers. In addition, newer 
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versions of the Windows OS actively thwart API intercepting methods described m the 
programnungUterature for the eariier versions of the OS. In some instances, a 
technological race is unfolding between the operating system developers and the third 
party (application) developers who need to provide OS/applications enhancements 
5 unforseen (or undeared) by the OS developers. 

A method for intercepting Windows 3 jc APIs based on patching the entry point of 
the API function with a JMPuistruction is known. In this system, the intercq)tor first 
obtains the address of the targrt API function via a caU to GetProcAddressQ. Note that 
this step has been actively thwarted by tiie OS in Win95 for many key API functions, 

10 alUiough wiUiin montfis the workarounds for much of the Uiwarting attempts have been 
published. FoUowing getting the procedure address, tije mterceptor removes the 
write-protection of Uie obtained memory address (whidi is a code address, thus h is set by 
the OS as read/execute-only). Following tfus, tiie interceptor patches tfie API entry point 
witii JMP InterceptSrv instruction, w^ere InterceptSrv is a service function in the 

15 interceptor's code that monitors and/or processes the API calls. 

After tiie interceptor has set up the API call in tiie manner described above, an 
appBcation or OS calls tiie API entry point. The JMP InterceptSrv instruction at tiie 
entry point address transfers control to tiie InterceptSrvO fimction and tiie InterceptSnO 
processes tiie call (accessing as necessary tiie fimction arguments on tiie stack) and. upon 

20 completion, dtiier returns control to the caller or passes control to tiie ori^nal API 

fimction. Returning control to the caller is straightforward. Pasang control to the API. 
on the otiier hand, requires that the InterceptSrvQ remove tiie patdied in JMP 
InterceptSrv from the API entry point and restore tiie ori^nal opcode bytes (tiie ori^nal 
five bytes tiiat were present in the API before tiie patch was inserted). Then, the 

25 InterceptSrvO pushes all original arguments fi^m tiie.caller*s stack on to the current stack 

Onterccptor-s stack). Finally, the InterceptSrvQ calls the original API fimction. When the 
ori^al API returns, the InterceptSrvQ saves the return value mto a local variable, the 
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InterceptSrvO reinserts the JMP InterceptSrv patch at the API entry point, and the 
InterceptSrvO returns the saved return value to the original caMer. 

The technique described above is used in many commercial appUcation enhancers 
However, it has many drawbacks. For one thing, the division of labor between the set up 
portion and the run time portion is highly inefficient since the set up portion is executed 
only once (at the interceptor's load time) while the tun time portion is executed many 
times (from the load time onward). In a more efficient system, as much of the work as 
possible would be shifted from the run time portion to the set up portion. In addition, for 
a processor running MS Windows on an Intel processor, the necessary replacement of the 
JMP Intercept opcodes requires a write operation of five bytes, which can not be done in 
a single processor instorction. Hence, the replacement leaves a short interval of instability 
between the two write instructions, during which the API entry point has invafid 
instructions and a hardware mterrupt at that moment could initiate reentry into the API, 
which will probably crash the system. 

On the ol her hand, disabling the interrupts in Windows appUcation mode (CPU 
ring 3 code where these actions are occurring) is a highly expensive operation due to 
system control over the CPU interrupt flag, which triggers an elaborate exception process 
Cm CPU ring 0). The complex ring transition and tiie exception process, which may take 
as much or more time tfian all the rest of processing above, would occur twice. Hierefore 
most commercial interceptors (as weU as tire published code) choose tire tradeoff with the 
instabffity allowed, in order not to pay the disproportionate performance cost associated 
with disabling interrupts. 

Anotiier disadvantage of the technique described above is that, during certain time 
intervals, another caU to the monitored API wiU be missed by the interceptor. Since many 
Windows APIs perform checks on the task queue within and can (and often do) switch to 
anotiter tiiread/task. the missing of intercepts is a real problem for intercepton^ which 
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require processing on ev«y caU to the API (especially those implementing security 
features). Also, the overwriting of the API entry point on eveiy call is unsafe when 
multiple interceptors exist on the same system. For example, while the system is 
processing the original API call, a switch to another task or thread can (and often will) 
occur. If the second task inserts its own intercept for the same API (or spawns a program 
which does that), then return to the first interceptor will destroy the new intercept, thus 
permanently disabling operation of the new intercept. Additionally, if tiie first intercept 
unloads, followed by an unload of second intercept, tiien the API entry wiU be left pointing 
to the non-existent first intercept and the system wffl crash when the API is invoked. 

Another disadvantage of the technique described above is that some Windows API 
functions (e.g. memory allocation and protection) are sensitive to tiie source of tfie call, 
Le. the Windows API code wiU check where the caD is made fi^om and, based on 
knowledge of Microsoft's sources of caUs, wiU woric differently if called fiom tiiird party 
appHcations as opposed to particular Microsoft sources. This behavior is, among other 
reasons, related to tiie active tiiwarting of the third party interceptors mentioned above. 
Since tiie intercept technique described above changes the ori^nal source of tiie API caH 
and makes tiie interceptor appear to Windows as the source of the API call, tiie Wmdows 
API processing may operate differently, often malfimctioning in a way tiiat leads to 
instabdities and system crashes. This makes tiie technique described above unsuitable for 
intercqpting some of the Windows APIs. 



Summary Of The Tnvftptl On 

Accordmg to the present invention, supplementing a software routine loaded in 
computer memory mcludes loading an additional routine into tiie computer memory, 
providing relocated opcodes by relocating a number of bytes fi-om a relocatable portion of 
the software routine to an otiier memory location where tiie number of bytes corresponds 
to an integral number of instructions of the relocatable portion, causing program control 
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to flow from the additional routine to the relocated opcodes, causing program control to 
flow from the relocated opcodes to a memoiy address immediately foUowing the 
relocatable portion, and causing program control to flow from the relocatable portion to 
the additional routine. 

5 Causing program control to flow from the additional routine to the relocated 

opcodes may include placing the relocated opcodes at a location in the computer memoiy 
that immediately foUows the additional routine. Causing program control to flow from the 
rdocated opcodes to the memory address immediately foUowing the relocatable portion 
may include placing a program control instruction at a location in the computer memoiy 
10 immediately fi)llowing the relocated opcodes. The program control instruction may be an 
unconditional jump instiuction. Causing program control to flow from the relocatable 
portion to the additional routme may include placing a program control instruction at a 
memory location corresponding to a source address of the relocatable portion of the 
software routine. The program control instruction may be an unconditional jump 
instruction. Providing tiie relocated opcodes may include relocating a number of bytes 
tiiat is at least equal to an amount of bytes required for tiie unconditional jump instmction. 



15 



FoUowing providing the relocated opcodes, it is possible to resolve any opcodes 
contained tfierdn that reference relative displacements between the relocated opcodes and 
opcodes contained in the software routine. The software routine and additional routine 
20 may be API's that run under the Microsoft Windows operating system. The additional 
routine may be configured to load at a predetermined address in the computer memoiy. 

According fiirther to the present invention, supplementing a Windows API loaded 
in computer memory includes loading an additional routine into the computer memory, 
providing relocated opcodes by relocating a number of bytes from a relocatable portion of 
25 the API to an other memory location where the number of bytes corresponds to an 

integral number of opcodes of tiie relocatable portion, causing program control to flow 
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fiom the additional routine to the relocated opcodes, causing program control to flow 
from the relocated opcodes to a memory address immediately following the relocatable 
portion, and causing program control to flow from the relocatable portion to the 
additional routine. According further to the present invention, a software program that 
supplements a Windows API loaded in computer memory includes an additional routine 
that is loaded into the computer memory, first means for providing relocated opcodes by 
rdocating a number of bytes from a relocatable portion of the API to an other memory 
location where the number of bytes corresponds to an integral number of instructions of 
the relocatable portion, second means, coupled to the first means and to the additional 
routin<^ for causing program control to flow from the additional routine to tiie relocated 
opcodes, third means, coupled to first means and to the API, for causing program control 
to flow frorii the relocated opcodes to a memory address immediately followng the 
relocatable portion, and fifth means, coupled to tiie API and to tiie additional routine, for 
causing program control to flow fix)m the relocatable portion to the additional routine. 

In the intercept instaU phase, Uie technique described herein disassembles tiie target 
API entry code and relocates (based on tiie semantics of the instructions found tfiere) the 
whole instructions from tiie API entry into the inteiwptor's memory. Then, in tiie intercept 
operation phase, when control needs to be passed to the original API fiinction, instead of 
having to swap back and fortii the overlayed API entry opcodes, tfie mtereept smiply 
passes control to the second (relocated) copy of tiie API entry code, which, upon 
completion, passes control to tiie next section of tiie original API code (tiie section wUch 
follows the relocated section). 

This method thus shifts the division of labor heavily toward the install phase of the 
intercept, relieving the active phase of tiie intercept by eliminating many of tiie steps 
described in connection witii the prior art system and the performance, safety and system 
stability drawbacks associated with them. 
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The technique described herein has many advantages over conventional trapping 
systems. One advantage is that all trapping work, except for a minimum amount of work 
necessary to transfer control to the interceptor and back, is done only once at install time, 
refieving the performance burden fix)m the nin time activity of the interceptor. In addition, 
no opcode swapping is done during the existence and activation of the traps. This 
eliminates performance, stability and safety drawbacks resulting fiom the activity found in 
some conventional systems. Furthermore, since the execution of the some of the trapping 
code occurs on the stack of the original caller, there is no need to copy API function 
aiguments to the interceptor's stack. By executing much of the code on the original 
callei's stack, the source of the call will appear to the Windows API as if it came from the 
original caller, therefore resolving problems associated with Windows behaving differently 
depending on the identity of the caller. By not copying API entry opcodes back and forth 
at each API call, no window of system instability is created. Instead, time-consuming 
precautions (e.g.. disabUng interrupts) occurs once at intercept install time. 

There are additional advantages. Since the trap opcodes are never removed during 
processing of the original API, the possibility of missing API calls is significantly 
decreased, as described above. Therefore, the system described herein is an exceUent 
choice for situations where the interceptor must see aU of the API calls to the target API 
function to operate properiy or reUably. The problem of multiple interceptors setting traps 
wMle processing of tiie trap is going on is resolved, since the new interceptor wiU always 
see fixed opcodes at the API entry point, tfius ihe second interceptor wOl not be forcibly 
disabled, as in conventional techniques. Also, since the settmg and removal of the traps 
occur only once at load/unload tune of intercqjtor, the problem of dangling intercept (with 
target of IMP InterceptSrv already unloaded) can be avoided since the interceptor can 
afford more detailed, time consuming, checks for safe removal of the intercepts (e.g. by 
refiising to unload itself if it is not tiie last interceptor). Doing these checks in the old 
trapping system is not only time consuming on every API call, but it is in most cases 
extremely difficult, if not inherently impossible, since it would require tiiat the interceptor 
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refose execution of the original API call, which will cause malfunction in tiie calling 
appUcation. In addition, tiie system described herein protects against unauthorized 
canceling of instaUed API security functions since canceling a new API task installed using 
the technique described herein, without restoring the original opcodes of the original API, 
5 will likely cause the system to crash. 

Brief Description Of Drawinp; 

FIG. 1 is a diagram illustrating a relationship between an old API and a new API 
according to the present invention. 

FIG. 2 is a flow chart showing steps that are performed to instafl the new API 
10 according to the present invention. 

FIG. 3 is a flow chart Ulustrating steps that are performed to remove the new API 
that is installed using the steps of FIG. 2. 

Petailed Description of th^ Pr..ferrftri Fmh>^in>>ri>fr!) 

Refer to HG. 1, a diagram 10 illustrates a relationship between an old API 12 and 
a new API 14. The old API 12 represents an existing API provided with an operating 
system, such as MS Windows 95. The new API14 represents an API tiiat is provided for 
use in connection with, for example, an appUcations program. As discussed in more detail 
below, the new API 14 can be executed instead of tiie old API12 or can be executed in 
addition to tiie old API 12. Note tiiat. in some instances, tiie new API 14 and/or the old 
20 API 12 may be referred to herem as a "routine". However, the term "routine" should not 
be understood as referring to a single, unitaiy. block of code but, instead, should be 
understood to refer to a collection of code tiiat may be provided in a pluraUty of blocks 
that may make calls or jumps tiierebetween. Note also tiiat the specific fiinctionaUty 
provided by the new API 14 is a design choice but may, in some instances, mcluding 
25 saving and restoring registers used by the caller and/or the old API 12. 
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In order to execute the new API 14, the old API 12 is patched with an 
unconditional jump instiuction 16 that transfers control from the old API 12 to the 
beginning of the new API 14. If the new API 14 is executed instead of the old API 12 ; 
0.e., the old API 12 is not to be executed), then a return to the callihg routine occurs at 
the end of the new API 14, as mdicated by the dotted line shown at the end of the new 
API 14. Note that, as will be apparent to one of ordinary skill m the art. other suitably 
equivalent control flow instructions may be used in place of the unconditional jump 
instruction 16. 

If the old API 12 wiU be executed Lnaddition to the new API 14, then opcodes 
that were located at a relocatable portion of the old API 12 Cm this case the beginning of 
the old API 12) become relocated code 18 that is placed at the end of the new API 14. As 
discussed in more detail hereinafter, any relative offsets between opcodes within the 
relocated code 18 and opcodes in the remainder of the old API 12 are adjusted, as 
appropriate. Note that, as wUl be apparent to one of ordinary sfciU m the art, it is possible 
to phice the relocated code 18 at an other portion of memory and then use an appropriate 
control flow instiuction at the end of the new API 14 to transfer program control from the 
new API 14 to the relocated code 1 8. 

Immediately foUowing the end of the relocated code 18 is an unconditional jump 
instrucdon 19 that transfers program control to the portion of the old API 12 immediately 

20 foUowing the relocatable portion of the old API 12. marked on the diagram 10 with the 

address "CONT". Thus, if both the old API 12 and the new API 14 are to be executed, 
then the calling routine calls the old API 12 which jumps, via the jump mstiuction 16, to 
the beginning of the new API 14 which then executes and, at the end thereof, executes the 
opcodes of the relocated code 18 foUowed by the jump instiuction 19 that jumps back to 

25 the remainder of the old API 12. Note that the relocated code 18 and the portion of the 

old API 12 beginning at the CONT address constitute the entu-ety of the old API 12. 
Also note that, as will be apparent to one of ordinary skill in the art. other suitably 
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equivalent control flow instructions may be used in place of the unconditional jump 
instmction 19. 



Referring to Fig. 2, a flow chart 20 illustrates steps for making patches that cause 
execution of the new API 14 when an appUcation program or the operating system calls 
5 the old API 12. Processing begins at a first step 22. where a byte from the beginnii« of 
the old API 12 is fetched. FoUowing the step 22 is a test step 24 which detennines if a 
whole instruction (as opposed to a partial instruction) has been fetched. This 
determination is made in a conventional fashion by. for example, disassembling the fetched 
bytes. Note that it is necessary to fetch an integral number of instructions from the 
10 relocatable portion of the old API 12 since it is not possible to execute a partial 
instruction. 

If it is determined at the test step 24 that one or more whole instructions have not 
been fetched, then control passes from the test step 24 back to the step 22 to fetch another 
byte. Otherwise, if an integral number of instructions have been fetched, then control 

15 passes from the test step 24 to a test step 26 wWch determines if enough bytes have been 

fetched from the relocatable portion of the old API 12 to accommodate the unconditional 
jump instruction 16 that transfers control from the old API 12 to the new API 14. In 
some embodiments, the required number of bytes is five. However, the test step 24 
preceding the test step 26 makes it posable that a number of bytes greater than five will 

20 have been fetched since it is necessary that a number of bytes corresponding to an integral 
number of instructions be fetched from the relocatable portion of the old API 12. 

If it is determined at the test step 26 that enough bytes have not been fetched, then 
control passes from the test step 26 back to the step 22 where another byte is fetched. 
Otherwise, if enough bytes have been fetched, then control passes from the test step 26 to 
25 a step 28. Note that it is not possible to reach the test step 26 without having fetched a 

number of bytes corresponding to an integral number of instructions. This is because it is 
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not possible to execute the test step 26 without having passed the test at the step 24. 
which detennines that the number of fetched bytes corresponds to a whole number of 
instnictions. 

At the step 28, the bytes that have been fetched are moved from the relocatable 
5 portion of the old API 12 to the end of the new API 14 and any opcodes in the relocated 
code 18 that refer to relative ofisets are resolved. Note that opcodes in the relocated code 
18 that contain a relative ofl&ct, such as a jump relative or a call relative, may need to be 
modified when the opcodes are relocated. Also note that, the relative positions within 
memory of the old API 12 and the new API 14 should not change after the APPs 12 14 
10 are loaded in memoiy. 

FoUowing the step 28 is a step 30 where the unconditional jump instruction 19 is 
added to the end of the new API 14. As discussed above, the unconditional jump 
instruction 19 causw control to return back to the portion of the old API 12 that foflows 
the relocatable portion of the old API 12. FoUowing the step 30 is a step 32 where the 
15 unconditional jump instruction 16 is added to the beginning of the old API 12 so tiiat 
vdien an appUcation program or U»e operating system caHs tiie old API 12, the 
unconditional jump instruction 1 6 fi-om the old API 1 2 to the new API 14 will be 
executed. 

As discussed above, it is possible timt the new API 14 entirely replaces the old API 
20 12 so that no part of the old API 12 needs to be executed once tfie new API 14 has been 
provided. In tiiat case, an alternative patch is provided. As shown in FIG. 2, control 
passes firom the test step 26 to a step 34 where a return instruction is added to the end of 
the new API 14 Of a return instruction is not already found at the end tiiereof). Following 
the step 34. control passes to tiie step 32 where tfie unconditional jump instruction 16 is 
25 added to tiie relocatable portion of tiie old API so that a call to the old API 12 wfll cause 
program control to flow fi-om the old API 12 to the new API 14. 
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Note that, in some instances* it may be unadvisable to relocate the code that b at 
the be^nning of the old API 12. For example, there may be other instructions wthin the 
old API 12 that reference the code located at the be^nning thereof In those cases, it is 
possible to use other portions of the old API 12 as the relocatable portion. For example, it 
would be possible to relocate bytes following the first N microprocessor instructions, in 
the manner described above, and replace the relocated bytes with the unconditional jump 
instruction 16. 

The code that executes the patching step illustrated by the flow chart 20 may be 
written in a conventional computer source language, such as C++, and compiled in a 
conventional manner similar to compilation of other Microsoft Windows DLL's, In some 
instances, the preferred base address of the new API 14 may be set to a vahie that will 
cause the new API14 to always load at the same address for all the processes whidi use 
the new API 14. Thus, the new API 1 4 may be shared so that the new API 14 is loaded in 
. memory only once, even when used by multiple processes. 

Refer to FIG. 3, a flow chart 40 Ulustrates steps that are performed v^en a process 
that uses the new API 14 is removed from memory. Note that, under the Windows 
environment, a special routine (MS Main) is called when a process is removed fiom 
memory. The MS Main routine provides the application with an opportunity to do 
cleanup including, in this instance, rratoring the old API 12. 

Processing begins at a first test step 42 where it is determined if the process being 
removed is the last process that uses the new API 14. If not, then the old API 12 and the 
new API 14 are not modified and no cleanup is done, since the new API 14 must remain 
to be used by the other processes. Otherwise, control passes fiom the test step 42 to a 
step 44 where the relative instructions of the relocated code 18 are modified back to the 
ori^nal state prior to restoring the relocated code into the old API 12. FoUowng the step 
44 b a step 46 where the relocated code 18 is restored into the old API 12, thus 
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overriding the unconditional jump instruction 16 that was provided to the old API 12 
when the old API 12 was patched. Once the relocated code 1 8 is restored to the old API 
12, then a call to the old API 12 wfll not result in execution of the new API 14. 

Note that, although the invention has been iUustrated herein using APIs with the 
Windows operating system, it would be straight-forward for one of ordinary sIdU in the art 
to adapt the system described herein to other operating systems and other types of 
routines. 

While the invention has been disclosed in connection wth the preferred 
embodiments shown and described in detail, various modifications and improvements 
thereon will become readUy apparent to those sldlled in the art Accordingly, the spirit 
and scope of the present invention is to be Bmited only by the foUowing claims. 
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Claitn(s) 

1 . A method of supplementing a software routine loaded in computer memory, 
comprising: 

(a) loading an additional routine into the computer memory; 

(b) providing relocated opcodes by relocating a number of bytes fiom a 
relocatable portion of the softvsmre routine to an other memory location, 
the number of bytes corresponding to an integral number of instructions of 
the relocatable portion; 

(c) cau^g program control to flow from the additional routine to the 
relocated opcodes; 

(d) causing program control to flow from the relocated opcodes to a memory 
address immediately following the relocatable portion; and 

(e) causing program control to flow from the relocatable portion to the 
additional routine. 

2. A method, according to claim 1, wherdn cau^g program control to flow from the 
additional routine to the rdocated opcodes includes placing the relocated opcodes at a 
location in the computer memory that immediately follows the additional routine. 

3. A method, according to claim 1, wherdn causing program control to flow from the 
relocated opcodes to the memory address immediately following the relocatable portion 
includes placing a program control instruction at a location in the computer memory 
immediatdy follomng the relocated opcodes. 

4. A method, according to claim 3, wherein the program control instruction is an 
unconditional jump instruction. 

5. A method, according to claim 1, wherein causing program control to flow from the 
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relocatable portion to the additional routine includes placing a program control instruction 
at a memory location corresponding to a source address of the relocatable portion of the 
software routine. 

6. A method, according to daim 5, wherein the program control instruction is an 
unconditional jump instruction. 

7. A method, according to claim 6, wherdn providing the rdocated opcodes includes 
rdocating a numb^ of bytes that is at least equal to an amount of bytes required for the 
unconditional jump instruction. 

8. A method, according to claim 1, further comprising: 

(f) following providing the relocated opcodes, resolving any opcodes 
contained therdn that reference relative displacements between the 
relocated opcodes and opcodes contained in the software routine. 

9. A meldod, according to claim 1, wherdn the software routine and additional routine are 
API's that run under the Microsoft Windows operating system, 

10. A method, according to claim 9, wherein the additional routine is configured to load at 
a predetermined address in the computer memory. 

1 1. A method of supplementing a Windows API loaded in computer memoiy, comprising: 

(a) loading an additional routine into the computer memory; 

(b) providing relocated opcodes by relocating a number of bytes from a 
relocatable portion of the API to an other memory location, the number of 
bytes corresponding to an integral number of instructions of the relocatable 
portion; 

(c) causing program control to flow from the additional routine to the 
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relocated opcodes; 

(d) causing program control to flow from the relocated opcodes to a memory 
address immediately following the relocatable portion; and 

(e) causing program control to flow from the relocatable portion to the 
additional routine. 

12. A software program that supplements a Windows API loaded in computer memory, 
comprising: 

an additional routine that is loaded into the computer memory; 

first means for pro>dding relocated opcodes by relocating a number of bytes from a 
relocatable portion of the API to an other memory location, the numbw of bytes 
corresponding to an integral number of instructions of the relocatable portion; 

second means, coupled to the first means and to the additional routine, for causing 
program control to flow from the additional routine to the relocated opcodes; 

third means, coupled to first means and to the API, for causing program control to 
flow from the relocated opcodes to a memory address immediately following the 
relocatable portion; and 

fifth means, coupled to the API and to the additional routine, for causing program 
control to flow from the relocatable portion to the additional routine. 
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